[ci] Support running re-execution benchmark with arbitrary version of Firewood #4650

Elvis339 · 2025-12-03T17:43:06Z

Why this should be merged

Enables Firewood to track performance over time by running C-Chain reexecution benchmarks with custom Firewood builds. This establishes the infrastructure for catching performance regressions before they reach production.

ava-labs/firewood#1494

How this works

Firewood triggers the existing C-Chain reexecution workflow via GitHub API with with-dependencies parameter
Workflow sets up dependencies via task setup-reexecution-deps before calling the benchmark action
The task uses polyrepo to sync Firewood at the specified ref
Runs C-Chain reexecution benchmark with the custom build
Uploads results as artifact for Firewood to download and track separately

The root action remains unaware of external dependency configuration - setup is handled by the workflow before invocation.

The same functionality is available locally for development:

nix develop
FIREWOOD_REF=abc123 ./scripts/run_task.sh setup-reexecution-deps
./scripts/run_task.sh test-cchain-reexecution -- firewood-101-250k

Changes

Add setup-reexecution-deps task for dependency setup (externalized from action)
Update workflows to call setup task before invoking benchmark action
Root action no longer accepts dependency refs (workflow responsibility)
Always upload benchmark artifacts (enables downstream tracking)
Update documentation in C-Chain Re-Execution README

How this was tested

gh workflow run "C-Chain Re-Execution Benchmark w/ Container"
--ref es/enable-firewood-dev-workflow
-f test=firewood-101-250k
-f with-dependencies="firewood=v0.0.15,libevm=v1.13.15-0.20251210210615-b8e76562a300"
-f runner=avalanche-avalanchego-runner-2ti
-f timeout-minutes=60

Need to be documented in RELEASES.md?

No

Copilot

Pull request overview

This PR establishes infrastructure for tracking Firewood performance over time by enabling C-Chain reexecution benchmarks with custom Firewood builds. The workflow can be triggered from the Firewood repository with either published versions (for quick testing) or branch/commit references (for comprehensive testing with source builds).

Key changes:

Adds a reusable workflow that accepts Firewood version/branch/commit as input
Implements intelligent build strategy: uses go get for published versions, builds from source for branches/commits
Creates build script for compiling Firewood FFI from source using Nix

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 2 comments.

File	Description
`.github/workflows/c-chain-reexecution-benchmark-firewood.yml`	New workflow that orchestrates benchmark execution with custom Firewood builds and uploads results as artifacts
`graft/coreth/scripts/build_firewood.sh`	Shell script to clone, build, and optionally integrate Firewood FFI from source
`.github/workflows/c-chain-reexecution-benchmark-container.yml`	Removes container configuration (unrelated cleanup)

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

graft/coreth/scripts/build_firewood.sh

.github/workflows/c-chain-reexecution-benchmark-container.yml

graft/coreth/scripts/build_firewood.sh

Taskfile.yml

.github/workflows/c-chain-reexecution-benchmark-firewood.yml

…/avalanchego into es/enable-firewood-dev-workflow

…ution benchmarks Integrate firewood/libevm dependency overrides into existing workflows using polyrepo, eliminating the need for a separate firewood workflow. - Add LIBEVM_REF/FIREWOOD_REF to reexecute-cchain-range-with-copied-data - Update composite action and workflows to pass inputs - Remove redundant firewood workflow and build_firewood.sh script

…/avalanchego into es/enable-firewood-dev-workflow

aaronbuchwald

I do not think that the re-execution benchmark action itself should be responsible for coupling firewood/libevm refs when any workflow invoking this can simply update the dependencies before invoking this action as a step. Separately, the action should not support a special case of archiving data just for Firewood.

These are simple pre- and post- execution steps, which can be easily added as a step before or after the action gets invoked within a workflow.

If Firewood wants to execute this separately, why not just define its own workflow to do so? If it wants to execute it within AvalancheGo, then we should just write the data to the default location using Firewood as the config.

…action inputs, add benchmark artifact upload step

…dencies for reexecution benchmarks

…nput handling for libevm, firewood refs, and refine artifact upload logic

Elvis339 · 2026-01-21T18:32:28Z

I do not think that the re-execution benchmark action itself should be responsible for coupling firewood/libevm refs when any workflow invoking this can simply update the dependencies before invoking this action as a step. Separately, the action should not support a special case of archiving data just for Firewood.

These are simple pre- and post- execution steps, which can be easily added as a step before or after the action gets invoked within a workflow.

If Firewood wants to execute this separately, why not just define its own workflow to do so? If it wants to execute it within AvalancheGo, then we should just write the data to the default location using Firewood as the config.

Resolved offline. Root action now:

Has no knowledge of firewood/libevm refs (workflow handles setup via task)
Pushes to GitHub Action Benchmark (controlled by input)
Always uploads artifacts

Dependency setup externalized to task setup-reexecution-deps, called by workflow before action.

maru-ava · 2026-01-22T16:10:55Z

Taskfile.yml


+  setup-reexecution-deps:
+    desc: Setup custom deps for reexecution benchmarks. Use LIBEVM_REF and/or FIREWOOD_REF env vars.
+    cmds:


(No action required) Maybe update the polyrepo script to accept LIBEVM_REF and FIREWOOD_REF so that this task is just providing those env vars to the script?

That's so much better, thanks for recommending this.

954121b

Edit: new commit, updated doc and how polyrepo is being ran 1f06a14

maru-ava · 2026-01-22T16:25:33Z

.github/workflows/c-chain-reexecution-benchmark-gh-native.yml

    runs-on: ${{ matrix.runner }}
    steps:
      - uses: actions/checkout@v4
+      - name: Install Nix


Maybe make nix installation optional as well?

Also, what is the implication of calling this action twice (once here, once in the reexec action)?

Made Nix installation conditional - only runs when libevm-ref or firewood-ref is specified.
Why workflow installs Nix before the action:

Polyrepo builds Firewood via nix build

The nix develop shell provides Go for go get libevm@

Double install implication: The action's Nix install is idempotent it detects the existing installation and skips:
https://github.com/ava-labs/avalanchego/actions/runs/21261969343/job/61191483101#step:6:53

The run failed but unrelated to Nix - the github-action-benchmark step tries to checkout gh-pages but fails because go.mod/go.sum were modified by the dependency setup.

Edit: fixed e3ab8ad https://github.com/ava-labs/avalanchego/actions/runs/21262297545/job/61192576708

Maybe add a comment documenting the idempotency for the benefit of future maintainers?

maru-ava · 2026-01-22T16:28:04Z

.github/actions/c-chain-reexecution-benchmark/action.yml

+      with:
+        name: benchmark-output-${{ inputs.run-id }}-${{ inputs.run-attempt }}-${{ inputs.job }}-${{ inputs.test }}-${{ inputs.runner_type }}
+        path: ${{ env.BENCHMARK_OUTPUT_FILE }}
+        retention-days: 30


Maybe omit this? afaik 30 days is the default

The default is 90 days per GitHub docs.

I'm keeping 30 days explicit because: (1) it documents intent rather than relying on assumed defaults, and (2) 30 days is enough for debugging while avoiding unnecessary artifact accumulation given our run volume.

While documenting intent might be a good principle, what is your intent in this case? We don't set retention days for other uses of this action, what makes this one special?

Fair point. Removing for consistency with other artifact uploads in the repo. 221157a

maru-ava · 2026-01-22T16:29:09Z

.github/actions/c-chain-reexecution-benchmark/action.yml

+    - name: Upload Benchmark Artifact
+      uses: actions/upload-artifact@v4
+      with:
+        name: benchmark-output-${{ inputs.run-id }}-${{ inputs.run-attempt }}-${{ inputs.job }}-${{ inputs.test }}-${{ inputs.runner_type }}


What's the rational for embedding all these values in the name?

Artifact names must be unique within a workflow run. Matrix jobs run in parallel and would otherwise conflict with a 409 error. These values ensure each matrix combination produces a uniquely named artifact.

The identifiers ensure uniqueness:

run-id + run-attempt → unique per workflow execution

job → matrix job identifier

test + runner_type → differentiates matrix combinations

Artifacts are already effectively namespaced by PR run (i.e. run-id and run-attempt). That's why we can use names like upgrade-tmpnet-data and not have collisions across PR runs.

Simplified the name. Thanks for explaining, the tests indeed pass.

6aef9df

maru-ava · 2026-01-22T16:30:46Z

tests/reexecute/c/README.md

+
+First, set up dependencies using the `setup-reexecution-deps` task, then run the benchmark:
+
+```bash


(No action required) Maybe include a single example with both versions and document that both or either one can be provided? Same comment for the CI invocation examples.

d9b2717

Edit: new commit, updated doc and how polyrepo is being ran 1f06a14

maru-ava · 2026-01-22T16:31:44Z

tests/reexecute/c/README.md

+./scripts/run_task.sh test-cchain-reexecution -- firewood-101-250k
+```
+
+### How It Works


(No action required) Maybe focus this doc additions on usage rather than duplicating implementation details?

d9b2717

Edit: new commit, updated doc and how polyrepo is being ran 1f06a14

- Add LIBEVM_REF and FIREWOOD_REF env var handling to run_polyrepo.sh - Remove setup-reexecution-deps task from Taskfile.yml - Update CI workflows to call run_polyrepo.sh directly - Delete setup_reexecution_deps.sh (no longer needed)

… uniqueness within workflow run)

…g permission issue by redirecting Nix's temp directory to /tmp which has proper permissions on ARC

- Remove FIREWOOD_REF env var handling from run_polyrepo.sh - LIBEVM_REF remains as env var (needs go get) - Firewood passed as polyrepo arg: sync firewood@<ref> - Clean up workflow steps with inline conditional - Update README with new usage patterns

The github-action-benchmark step was running even when push-github-action-benchmark was false, causing failures when custom deps modify go.mod/go.sum (can't checkout gh-pages with uncommitted changes). Now the step only runs when comparison is actually needed.

- Comparison step conditional on push-github-action-benchmark - Default true preserves existing behavior - Prevents gh-pages checkout failure when go.mod/go.sum are dirty (i.e., polyrepo firewood setup)

When custom dependencies (firewood-ref or libevm-ref) are used, go.mod/go.sum are modified. This causes github-action-benchmark to fail when checking out gh-pages. New input skips comparison step only when custom deps are present, preserving summary display for normal PR runs.

Elvis339 · 2026-01-22T19:52:02Z

.github/actions/c-chain-reexecution-benchmark/action.yml

  push-github-action-benchmark:
    description: 'Whether to push the benchmark result to GitHub.'
    required: true
+  skip-benchmark-comparison:


When custom dependencies (firewood-ref or libevm-ref) are used,
go.mod/go.sum are modified. This causes github-action-benchmark
to fail when checking out gh-pages.

New input skips comparison step only when custom deps are present,
preserving summary display for normal PR runs.

Would it make sense to make this skip automatic with an appropriate warning if the tree is dirty?

maru-ava · 2026-01-22T21:05:53Z

.github/actions/c-chain-reexecution-benchmark/action.yml

  push-github-action-benchmark:
    description: 'Whether to push the benchmark result to GitHub.'
    required: true
+  skip-benchmark-comparison:


Would it make sense to make this skip automatic with an appropriate warning if the tree is dirty?

maru-ava · 2026-01-22T21:07:08Z

.github/workflows/c-chain-reexecution-benchmark-gh-native.yml

    runs-on: ${{ matrix.runner }}
    steps:
      - uses: actions/checkout@v4
+      - name: Install Nix


Maybe add a comment documenting the idempotency for the benefit of future maintainers?

maru-ava · 2026-01-22T21:09:15Z

Taskfile.yml

    desc: Runs shellcheck to check sanity of shell scripts
    cmd: ./scripts/shellcheck.sh

+  polyrepo:


Please use an action name e.g. run-polyrepo.

maru-ava · 2026-01-22T21:11:43Z

scripts/run_polyrepo.sh

+#   ./scripts/run_polyrepo.sh [polyrepo args...]
+#
+# Environment variables (optional):
+#   LIBEVM_REF - Git ref for libevm (runs: go get && go mod tidy)


Maybe use env vars for both args so that the caller has a consistent way of specifying both values?

ci(c-chain-reexecution-firewood)

d0155dd

Elvis339 self-assigned this Dec 3, 2025

Copilot AI review requested due to automatic review settings December 3, 2025 17:43

Elvis339 requested review from a team and aaronbuchwald as code owners December 3, 2025 17:43

Elvis339 added the ci This focuses on changes to the CI process label Dec 3, 2025

github-project-automation bot added this to avalanchego Dec 3, 2025

Elvis339 mentioned this pull request Dec 3, 2025

Track Firewood Performance via AvalancheGo Reexecution Benchmarks ava-labs/firewood#1494

Open

Copilot AI reviewed Dec 3, 2025

View reviewed changes

graft/coreth/scripts/build_firewood.sh Outdated Show resolved Hide resolved

graft/coreth/scripts/build_firewood.sh Outdated Show resolved Hide resolved

lint

e851903

Elvis339 requested review from joshua-kim and maru-ava as code owners December 3, 2025 18:02

Elvis339 commented Dec 3, 2025

View reviewed changes

.github/workflows/c-chain-reexecution-benchmark-container.yml Show resolved Hide resolved

Merge branch 'master' into es/enable-firewood-dev-workflow

49b7fc3

Elvis339 mentioned this pull request Dec 4, 2025

ci(perf): Track Firewood Performance via AvalancheGo Benchmarks ava-labs/firewood#1493

Draft

RodrigoVillar reviewed Dec 5, 2025

View reviewed changes

graft/coreth/scripts/build_firewood.sh Outdated Show resolved Hide resolved

Merge branch 'master' into es/enable-firewood-dev-workflow

2feeda4

Elvis339 requested a review from RodrigoVillar December 8, 2025 16:23

maru-ava reviewed Dec 9, 2025

View reviewed changes

Elvis339 added 4 commits December 9, 2025 19:45

ci(firewood-benchmark): try go get first, fall back to Nix on failure

6f21231

Merge branch 'es/enable-firewood-dev-workflow' of github.com:ava-labs…

b3cc8ed

…/avalanchego into es/enable-firewood-dev-workflow

refactor(build-firewood): move default config to script, simplify task

e437658

ci(firewood-benchmark): test workflow without the commit step

15b52ee

Elvis339 requested a review from maru-ava December 9, 2025 18:10

Elvis339 and others added 4 commits December 9, 2025 22:10

Merge branch 'master' into es/enable-firewood-dev-workflow

999c3f4

Merge branch 'master' into es/enable-firewood-dev-workflow

34a331f

Merge branch 'es/enable-firewood-dev-workflow' of github.com:ava-labs…

e0e784b

…/avalanchego into es/enable-firewood-dev-workflow

Elvis339 requested a review from StephenButtolph as a code owner December 12, 2025 17:12

fix(c-chain-reexecution): add nix build env vars for self-hosted runners

cab333e

joshua-kim removed their request for review January 6, 2026 15:25

Elvis339 and others added 3 commits January 6, 2026 20:15

Merge branch 'master' into es/enable-firewood-dev-workflow

2b86559

Merge branch 'es/enable-firewood-dev-workflow' of github.com:ava-labs…

90fd09a

…/avalanchego into es/enable-firewood-dev-workflow

Merge branch 'master' into es/enable-firewood-dev-workflow

73c876e

aaronbuchwald reviewed Jan 6, 2026

View reviewed changes

Elvis339 added 5 commits January 21, 2026 19:14

ci(c-chain-reexecution): remove unused libevm and firewood refs from …

a92e465

…action inputs, add benchmark artifact upload step

chore(Taskfile): add setup-reexecution-deps task to configure depen…

81b88b6

…dencies for reexecution benchmarks

ci(c-chain-reexecution): add setup-reexecution-deps step, improve i…

6f944d1

…nput handling for libevm, firewood refs, and refine artifact upload logic

docs

8a92625

Merge branch 'master' into es/enable-firewood-dev-workflow

db41178

Elvis339 requested review from RodrigoVillar and aaronbuchwald January 21, 2026 18:40

Elvis339 added 2 commits January 21, 2026 19:42

ci(c-chain-reexecution): unique artifact upload name

f410444

ci(c-chain-reexecution): unique name

d619a21

maru-ava reviewed Jan 22, 2026

View reviewed changes

Elvis339 added 12 commits January 22, 2026 19:17

docs:

d9b2717

refactor(ci): simplify dependency configuration for benchmarks

6076c4f

fix(ci): Simplify artifact name to test + runner_type (sufficient for…

221157a

… uniqueness within workflow run)

fix(ci): Simplify artifact name to test + runner_type (sufficient for…

6aef9df

… uniqueness within workflow run)

lint

c28cd33

fix(reexecution-container): add tmpdir and nix build top to tmp fixin…

2ac5157

…g permission issue by redirecting Nix's temp directory to /tmp which has proper permissions on ARC

lint

6ad2f13

fix(ci): skip benchmark comparison when custom deps modify go.mod

01da31a

- Comparison step conditional on push-github-action-benchmark - Default true preserves existing behavior - Prevents gh-pages checkout failure when go.mod/go.sum are dirty (i.e., polyrepo firewood setup)

Elvis339 commented Jan 22, 2026

View reviewed changes

maru-ava reviewed Jan 22, 2026

View reviewed changes


		First, set up dependencies using the `setup-reexecution-deps` task, then run the benchmark:

		```bash

[ci] Support running re-execution benchmark with arbitrary version of Firewood #4650

Are you sure you want to change the base?

[ci] Support running re-execution benchmark with arbitrary version of Firewood #4650

Conversation

Elvis339 commented Dec 3, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Why this should be merged

How this works

Changes

How this was tested

Need to be documented in RELEASES.md?

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

aaronbuchwald left a comment

Choose a reason for hiding this comment

Uh oh!

Elvis339 commented Jan 21, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Elvis339 Jan 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Elvis339 Jan 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Elvis339 Jan 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Elvis339 Jan 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Elvis339 Jan 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Elvis339 commented Dec 3, 2025 •

edited

Loading

Elvis339 Jan 22, 2026 •

edited

Loading

Elvis339 Jan 22, 2026 •

edited

Loading

Elvis339 Jan 22, 2026 •

edited

Loading

Elvis339 Jan 22, 2026 •

edited

Loading

Elvis339 Jan 22, 2026 •

edited

Loading